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Abstract 

We perform a stationary state replica analysis for a layered network of Ising spin neurons, 
with recurrent Hebbian interactions within each layer, in combination with strictly feed- 
forward Hebbian interactions between successive layers. This model interpolates between 
the fully recurrent and symmetric attractor network studied by Amit el al, and the strictly 
feed-forward attractor network studied by Domany et al. Due to the absence of detailed 
balance, it is as yet solvable only in the zero temperature limit. The built-in competition 
between two qualitatively different modes of operation, feed-forward (ergodic within layers) 
versus recurrent (non-ergodic within layers), is found to induce interesting phase transitions. 
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1 Introduction 



There are two main classes of solvable models for neural networks, functioning as associative 
information processing devices. The first class consists of models in which the matrix of in- 
teractions between the neurons is symmetric ||, ^j, f|. This ensures detailed balance, so that 
equilibrium statistical mechanics applies. The second class of models are equipped with neuron 
interactions which are non-symmetric, but are of a strictly feed-forward nature. Such models 
can be constructed explicitly, by arranging the neurons in feed-forward layers [|], |5|, ||], or im- 
plicitly, by introducing an extreme dilution such that on finite time-scales the systems behave 
effectively tree- like [|| . The latter models can no longer be studied within equilibrium statistical 
mechanics; here it is the strict feed-forward nature of the interactions which enables a dynamical 
solution. Even the more recent studies of layered attractor networks all rely for solution on the 
connectivity being either strictly symmetric (e.g. [§, ||]) or strictly feed-forward (e.g. |i~0|]). 

In this paper we study a dual model, where there is a controllable competition between 
the highly non-ergodic recurrent information processing, typical for symmetric systems, and the 
feed-forward processing typical for tree-like structures, where individual layers are ergodic. We 
consider chains of equally large neural layers, each consisting of Ising spin neurons with recurrent 
interactions, in combination with strictly feed-forward interactions between succesive layers. 
All allowed interactions are of a Hebbian type; this choice guarantees meaningful information 
processing, induces familiar order parameters, and creates convenient benchmarks, in that in the 
two extreme limits of fully recurrent and fully feed-forward connectivity the familiar results of 
0, [| and j|, [|, [| must be recovered. Each individual layer can be seen as a recurrent attractor 
network with association cues provided by (thermally fluctuating) external fields. These fields, 
however, contain contributions from non-nominated patterns, and therefore contain the disorder 
variables to be averaged out (which already appear in the recurrent interactions), in contrast to 



the situation with most other attractor models with external fields, such as pT| , 12, 13, [14J. 

Our motivation for studying this type of model is threefold. From a statistical mechanical 
point of view the model is of interest simply because its interaction matrix is neither symmetric 
nor strictly feed-forward, so that the conventional routes to a solution cannot be followed, and 
since it interpolates between two systems with quite different properties. Secondly, because the 
model can nonetheless be solved (at least in the zero temperature limit), it could serve as a 
convenient future playground for testing new formalisms aimed at calulating stationary state 
properties of non-symmetric neural network models, such as Jl5[ . Thirdly, from a biological point 
of view, both recurrent and feed-forward information processing have specific advantages and 
disadvantages, and especially in the peripherical brain regions one therefore usually finds both 
interaction types present. Recent experimental evidence even suggests that the balance between 
recurrent and feed-forward processing in these regions is actively controlled by neuromodulators 
[16]. Yet, as far as we are aware, no dual models have been solved so far. 

This paper is organised as follows. After having defined our model, we analyse its zero tem- 
perature stationary state, using replica theory. The resulting saddle-point equations are solved 
upon making the replica symmetry (RS) ansatz, leaving two control parameters: a (the informa- 
tion storage level) and u (a parameter reflecting the balance between recurrent and feed-forward 
operation). We then study the various types of phase transitions exhibited by the model: (i) 
the saturation (or storage capacity) transition, (u) simple (RS) ergodicity breaking transitions, 
and {Hi) complex (i.e. replica symmetry breaking, RSB) ergodicity-breaking transitions. All 
analytical results are confirmed by numerical simulations. 
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2 Model Definition and Solution 



2.1 Model Definition 

Our model system is a feed- forward chain of L recurrent layers, each of which consists of N 
Ising spin neurons. The variable af £ {—1, 1} denotes the state (non- firing/firing) of neuron i 
in layer £, and the collective state of the N neurons in layer £ is written as a 1 = (a{, . . . , c^ N ). 
The dynamics is the usual Glauber- type sequential stochastic alignment of the neurons a\ to 
local fields hf, to which here only neurons from within layer £ (via recurrent interactions) and 
neurons from the previous layer £ — 1 (via feed-forward interactions) are allowed to contribute 
(see figure [l]) : 

Probtf -^f] = ^[l-tanh^^V*- 1 ))] ^(*V _1 ) =E4^+E^" 1 « 

j j 

At each time step the candidate neuron (i, £) to be updated is drawn at random from {1, . . . , iV}x 
{1, . . . , L}. The parameter /3 = T _1 controls the amount of noise, with T = corresponding to 
deterministic alignment (although the order of updates still remains random). 




Layer £—1 Layer £ 



Figure 1: Two adjacent modules in the L-layer chain, with their associated microscopic vari- 
ables and parameters: a 1 G { — 1,1}^ (neuron states in layer £), {Jfj} (symmetric recurrent 
interactions in layer £), and {W^} (feed-forward interactions from layer £—1 to layer £). 

The 2L x iV 2 synaptic interactions are defined as the result of the network having learned p = aN 
patterns with a Hebbian-type rule: 

4 = It 1 -^ f <=if ^^r 1 ( 2 ) 
fj.=i 

with G {—1, 1} denoting component i in layer £ of pattern \i (fj, = 1, . . . ,p). Each pattern 
represents a specific microscopic neural configuration in the chain as a whole. For simplicity all 
pattern components are drawn independently at random from {—1, 1}, and N is assumed to be 
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large (eventually we will take the limit N — ► oo). The two parameters (Jo, J) (one of which will 
become redundant in the limit T — > 0) control the relative strength of the two interaction types. 
The only exception to (Q) is the first layer, where by definition we have to set W-j = (Vr/), 
and where we have to distinguish between two modes of operation: (i) free relaxation of layer 
1, following a specific initialisation which serves as the recall cue, and (ii) so-called 'clamped' 
operation, where the recall cue is provided by the state vector a 1 itself, which is stationary and 
specified externally. 

For J = our model reduces to a collection of L decoupled symmetric attractor networks of 
the Hopfield Q type, which can be solved in equilibrium using equilibrium statistical mechanics 
0, pi. Such systems are known to have a storage capacity of a c ~ 0.138 (for T = 0, in RS 
approximation), and non-trivial ergodicity breaking (RSB) for sufficiently low temperatures. 
For Jo = 0, on the other hand, we have feed-forward interactions only. Now the local fields are 
found to have Gaussian probability distributions in the stationary state, enabling the derivation 
of recurrent relations for the values of order parameters in subsequent layers 0, [|. Here 
one finds a storage capacity of a c ~ 0.269 (for T = 0) and no RSB at any temperature. The 
solution of the present model will have to represent a marriage of these two extremes, such that 
the equations first derived in and 0] follow as special cases, upon taking the limits J — > 
and Jo — > 0, respectively. 

Away from saturation, for a = p/N — ► 0, the dynamics of this model can be solved easily 
using existing techniques (for a review see [[ItJ ) , leading to a set of coupled differential equations 
for a small number of macroscopic observables. In order to solve the model near saturation (i.e. 
for a > 0), we will exploit the fact that, due to the strictly feed-forward nature of the inter-layer 
interactions, at T = all layers will eventually go to a stationary state; for each layer I there 
will be a stage after which the input from layer ^ — 1 is stationary. This enables an equilibrium 
statistical mechanical replica analysis at least in the T —* limit (for T > one has to take 
thermal fluctuations in the external fields into account, which destroy the Boltzmann form of 
the stationary states of the individual layers) . For technical reasons we will take the limit T — > 
after the limit N — ► oo, a commonly made step which in the present case, however, need not be 
as harmless as for models obeying detailed balance (it is justified a posteriori by the agreement 
between theory and numerical simulations). 



2.2 Replica Analysis of the Stationary State 

We analyse the stationary state for a given layer I, upon assuming stationary inputs from layer 
£—1. Given this assumption, the dynamics (|l|) will evolve towards equilibrium, characterised by 

fto^) ~ e-^) H{o>) = -\ £ 0"! jfj o~j - £ a'Ul^ 1 (3) 



The associated thermal averages are written as (•••)■ We introduce a macroscopic description 
in terms of the so-called overlaps between the system state and the stored patterns, and make 
the usual ansatz that in equilibrium only a finite number k of the patterns are condensed, which 
(due to permutation symmetry with respect to pattern indices) we can take to be \x = 1, . . . , k: 



1 



M €{!,...,*}: (m* (<r«)> = 0(1) 



m,M) = ^e4 < _ x . , :\ ;„ _ (4) 



fi€{k+l,...,p}: {mlM))=0{l/N) 
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We assume the free energy per spin / = — log Z of the system described by (J3j) to be self- 
averaging for N — > oo with respect to the realisation of the non-condensed patterns fi > k (which 
play the role of 'frozen disorder'), so that we can simplify the calculation of the free energy by 
averaging over these patterns. This property can also be rigorously proven. To simplify the 
pattern average we use the identity log Z = lim n ^o h [Z n — 1] , giving 



Z 



E 



-f3H(a) 



(5) 



cre{-i,X} J 



Upon writing the pattern overlaps of the previous layer t—\ (which are stationary by assumption) 
as rhfj, = X(J \ X -> we can write the key quantity in (||) for integer values of n in the 

following form: 

) P att 



e -±af3J n 



^2 ... Yl (e ,JiV ^,i:Li[i^,,(T a )+K,i(T a )%] 



patt 



(6) 

The condensed contribution to (|6|) (the terms \i < k in the summation over pattern indices) is 
linearised by a Gaussian transformation: 



/ JV E <1 < fc E«[l J " m M( " a )+- 7A M^M(o- 



[3J N 



2vr 



UU dm 

fj,<k a 



(7) 



The uncondensed contribution to @ (the terms \i > k in the summation over pattern indices) 
is first linearised by a Gaussian transformation and then averaged over the frozen disorder, i.e. 
the pattern components with \i > k, giving 



J> N E M > fc E Q [¥° m l (<T a )+Jm M m M/ (o-)] 



p att ti>k 



with the Gaussian measure Dz = \\ a (2tt) 2e 2 z &dz Q . Using the property fh?^ = 0(N 1 ) for 



fjL > k, we expand log cosh (x) 



1^2 



+ 0(x 4 ). Inserting an integral representation of unity to 



isolate the spin glass order parameters q a p = i J2i a ? a i > an d repeatedly forgetting about terms 
which are of vanishing order either for n — * or for N — > oo, then leads to: 



patt 



dqdq ^ a0 ^ H a/3+ ^ (/3J)2 ^M>fe ™l 



'E^ 9«s Ei CT f 



Tl [Dz e \PJoZ-qz+Y Ja z a x>i 



dqdq e 



lap 



^ + 5(/3J) 2 E M>fc " l M 



,; E Q(3 E t of-gplogdet A+±£ Q/3 E„>fc ^( A )a^ 



(8) 

with the abbreviation = @J{f3JoN) 1 / 2 rh lx Yl l 3 ( la(3i an d where A is an n x n matrix with 
components k a p = 5 a p — fiJ§q a p- We finally substitute the results (0,||) into equation (H|), and 
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perform the remaining summations over the spin variables {c"}- The free energy per spin (||) 
is subsequently obtained by interchanging the limits ./V — > oo and n — > and by performing the 
remaining integral by the method of steepest descent, leading to the final result: 

/ = ^aJ - /T 1 log 2 - hm — extr I iJ2q a pq a/ 3 - ^/3</oE E t 771 ^ 2 ~ \ a lo S det I 1- / 3 J o<?] 



aft"/ 



(9) 

with £ G {—1, l} fc and <r G {—1, l} n . The extremum in (|9|) refers to variation of the parameters 
{qa/3, Qa/3, ^^}) and is defined as the analytical continuation for n — > of the saddle point 
which for n > 1 minimises /. The physical meaning of the parameters can be deduced 
for instance by taking the derivative of (||) with respect to fh^, which gives lim n ^o ^ J2 a m ^ = 

2.3 Replica-Symmetric Solution 

We have obtained an expression for the free energy per neuron / in terms of the parameters 
Qa/3 , Qa/3 and m^. In order to take the limit n — » in (|9|) and obtain an explicit solution, we will 
make the replica symmetry (RS) ansatz for the relevant saddle-point, i.e.: 
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q a p = -jiafi 2 [R5 a p + r(l - 6 a p)] 
q a f3 = 5 aj 3 + q(l - 5 a p) 



(10) 



m" = m. 



Substitution of ([H]) into (|]) and linearisation of the exponent involving the spin variables, allows 
us to perform the remaining spin summations and take the limit n — > 0, which gives: 



fas = \a [Jo+/3r(l - q)] + ^J E m l + 

/i<k 



a 



log[l-/?J (l-g)] 



PJ 



l-/3J (l-q) 



l r v- ~ 2l /?</ 2 (l-<?) 



2^^1-/?J (l-g) /? 



- ( / L>zlog2cosh/5 



E ^( J o m v + Jrh v) + 



zwar 



fill 



where -Dz = (27r) ae ^eiz and = 2 fc X^ e {_i ip Taking the derivative of the 

free energy with respect to the control parameter Jq in addition yields the following identity: 



E KM)) = a - E KM)) - 2 



dfos 
dJ 



(12) 



By solving the saddle point equations df^s/dm^ = dfes/dq = dfes/dr = 0, from which we 
can eliminate ^] >(i .m^ by applying ( |i~2| ) to layer £ — 1, we finally arrive at a set of recurrent 
equations which relate the values of order parameters in subsequent layers. Upon adapting 



6 



our notation accordingly, these recurrent relations can be written in their most natural form 
(m,q,r) — > (m',q',r'): 



q = (J Dz tanh 2 (5 £ ■ (Jom' +Jm)+zV ar' 
r > [i.^i.g')] 2 . J V = j3 2 J 2 r(l-q) 2 -J 2 



J 2 (l + q) 



m' = (£ I Dz tanh/3 \£ ■ (J rri + Jm) + zVm J \)f (13) 

(14) 

l-^Jo(l-g) (15) 
The macroscopic state of every layer t is now characterised by its value for the order parameter set 
(m, q, r), with m = (mi, . . . , m^), and with the (familiar) physical meaning m M = -k J2i £f ) 
and q = jj Ei(°f} 2 - The only exceptions to ( |13[{T5| ) arise in the context of the special status of 
the first layer. If this first layer is clamped into a randomly selected configuration a 1 with a 
prescibed vector of the condensed overlaps m, then (|i~5|) is to be replaced by 



(16) 



[l- / 3J (l-<t)] i -J*j = J* 



If, on the other hand, the first layer is allowed free relaxation towards equilibrium, then ( |13| - 
15) does hold, but with the macroscopic state (m,q,r) of the first layer solved from the set 
corresponding to the J = situation, where inter-layer interactions are absent: 



m = (£ J Dz tanh/? [Jo£ • m+z^/ar])^ 
q = (J Dz tanh 2 /3 [Jo£ • m+zy/ar\)g 



^o 2 



(17) 
(18) 
(19) 

[l-pj (l-q)Y 

In the limit T — > 0, the equations (13-^) constitute the solution of our model. We will now 
analyse their consequences and validate their predictions with simulation experiments. We 
eliminate the parameter redundancy at T = by putting Jo = and J = i[l— w], with 

u€\-l,l]. 

3 Phase Transitions 

3.1 Saturation Transition in Infinitely Long Chains 

We first calculate the information storage capacity a c for (infinitely) long chains. A stationary 
situation is reached along the chain when (m,q,r) = (m',q',r') in (|l~3[jl5|), giving 



(20) 
(21) 
(22) 



m = (£ J Dz tanh (3 [£ • m+zy/ar] ) g 

q = ( J Dz tanh 2 (3 [£ • m+zy/ar\ )^ 

{l-Lo) 2 + q{l+uo) 2 -2uj(3q{l+oo){l-q) 
4[l-J/3(l+ w )(l-g)] [l-f3(l+cu)(l-q)+cuf3 2 (l-q) 2 ] 
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20 40 

layer number 



60 



Figure 2: Left picture - theory: storage capacity a c for long chains at T = (dashed: location of 
the optimum). Right picture - numerical simulations (N = 900): stationary (pure state) overlap 
m as a function of layer number £, for T = and u = 0. 



Upon taking the limit T — > and concentrating on pure states, where m. 



mS, 



M A, we can 



perform the averages and integrations, substitute x = m/y/2ar, and simply follow the procedure 
described in || to reduce the above set of equations (2C-^) to a single transcendental equation 
(with m = erf(x)): 



xv 2a 



erf(x) - ^e~ x ( [erf(x) - e 



{ [erf(x) - %e 



erf (x) 



Q+tJ)x — X Q 
^ 6 



erf (x) 



u) 2 +u) 2x - 



(23) 



This equation is to be solved numerically. The storage capacity a c is the value for a for which 
the nonzero solutions of ( p3| ) vanish, resulting in figure |2| (left picture). For u = — 1 equation 
(|23|) reduces to the results of ||, g]; for a; = 1 it reduces to the results of ||, as it should. 
Somewhat surprisingly, the largest storage capacity is obtained for u> ~ —0.12, giving a c ~ 0.317. 
We support these analytical results with numerical simulations on chains with L = 60 layers 
of iV = 900 neurons each, for T = and to = 0. Figure [2] (right picture) shows the stationary 
values of the condensed overlap m as a function of the layer number. For a E {0.26, 0.28} the 
desired state m ~ 1 appears stable, for a G {0.33, 0.35} the state m ~ 1 is unstable, whereas for 
a £ {0.29, 0.30} we appear to be close to the critical value, with m ~ 1 appearing stable initially, 
but eventually destabilising further down along the chain. This is in reasonable agreement with 
the theory, which predicts a c ~ 0.314 for u = 0, if we take finite size effects into account. 



3.2 Simple Ergodicity Breaking Transitions 

Next we turn to transitions marking the appearance of multiple (replica symmetric) coexistent 
stable states. We restrict ourselves to pure states, i.e. in each layer m M = m8^\ for some A 



and some layer-dependent m, and to the junction between layer i = 1 and layer I = 2, where 
the first such transitions are expected. Insertion of pure state solutions, and taking the limit 
T — > in ( |13| - [T5D again allows us to perform the averages and integrations. Substitution of 
y = [ Jam' + Jm] / \J 2ar' and, in the case where the first layer evolves freely, of x = Jqiti/ ' \J2olt , 
now leads to the expression m! = erf(y), in which y is the solution of the following transcendental 
equation: 



F(y) = yV2a 



1- 



-P 



1-oj 

1 + OJ 



in 



1-LO 



1 + CO 



eri (y) — i= e 

\/7T 



(24) 



The value of p in (24) depends on whether or not the input layer is clamped into a state, and, 
in the case of free relaxation, on the actual value of m: 



input clamped : 
free relaxation : 



m given, p = 1 

to = erf(x), F(x) = x\^2a, p = [erf (x)/ F(x)Y 



(25) 



The simplest situation, to = 0, already clearly demonstrates the distinction between the two 
modes of operation (input clamped versus free relaxation). In the clamped case, where p = 1, 
a rescaling of the storage ratio a maps ( |24"D onto the equations describing the model of , and 
solutions with y ^ (i.e. to' ^ 0) must therefore bifurcate at 



a bif (u;) 



«bif(l) 



0.138 



(26) 



On the other hand, for free relaxation of the first layer we find p = oo, so y = (i.e. to' = 
is the only solution of (^)). In the first case the imposed to = state of layer one is like a 
paramagnetic state; in the second case the m = state is of a spin-glass type. 



For arbitrary to, we obtain the condition for new solutions of (24) to bifurcate by derivation 
of ( p4| ) with respect to y. This gives a new equation, to be solved simultaneously with (p4[); a 
simple transformation allows us to rewrite the resulting pair as 

-i 



a 



u 4 

-y e 

IT 



-V 



1-u 

1+U) 



m 



1-u 



G{y) 



(27) 



(to be solved with the appropriate values for (m,p), given in (25)), and with 

erf(y) 



2 y „ , o ,2^-2,2 



G{y) = -j={l+2y')e 



(28) 

y vi 

It is clear from ( p7|) that bifurcations of new solutions can only occur for sufficiently small a. For 
clamped input operation, where the input overlap m is an independent parameter, numerical 
solution of (|27| ) results in bifurcation lines in the (w, a) plane, shown in figure ^. Left and above 
the solid lines are the regions with a unique stable state, at the other side of the solid lines (near 
uj = 1) multiple stable states exist. The bottom-right picture shows the result of combining all 
regions with multiple stable states, for m € [0, 1]: left of the solid line there is a single stable 
state in the second layer, irrespective of to, right of the solid line input overlaps to E [0, 1] can 
be found which give rise to multiple stable states in the second layer. These results are again 
supported by numerical simulations: figure |I| shows the relation between initial and (almost) 
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Figure 3: Examples of bifurcation lines, marking the appearance of multiple pure (and replica- 
symmetric) stable states in layer 2, for clamped operation. Left of and above the solid lines: a 
single stable state; close to the u> = 1 line: multiple stable states. Dashed lines: the long-chain 
storage capacity a c . Dots in m = 1.00 picture: control parameters (u>,a) used in numerical 
simulations (described below). Bottom-right picture: boundary of the cumulative region where 
multiple stable states exist (obtained by combining the results for input overlaps m £ [0, 1]). 
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a = 0.01 



a = O.Ol 
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-l 
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i 
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a = 0.20 



. ■ ■ j 



.5 

m'(t = 0) 



1 -1 



.5 .5 

m'(t = 0) 



Figure 4: Simulation results (N = 12,000), indicating the number of (pure) stable states in 
layer 2. Each picture shows the overlap m! at time t = 200 versus the overlap m' at t = 0, for 
uj = 0.9 and input overlap m = 1. The a values correspond to the dots in figure ||. 



final overlaps in layer 2, for N = 12,000 and m = 1, and the four (ui,a) combinations indicated 
with dots in figure |||. The number of stable states predicted (figure^) are (2, 3, 2, 1), for the four 
cases a = (0.01,0.08,0.14,0.20), respectively. This is in perfect agreement with the simulation 
results of figure ^j, where for each graph the number of stable states is the number of disconti- 
nuities plus one. For a system in equilibrium one expects horizontal line segments between the 
discontinuities; apparently for a G {0.08,0.14,0.20} the state at t = 200 is not an equilibrium 
state yet (due to the large relaxation times involved). 

For free relaxation of the first layer, a nonzero m, being a solution of (|25|), is a monotonically 
decreasing function of a with m(a) > 0.966. The bifurcation lines obtained by solving numer- 
ically (|25|,^7|) are now found to be practically indistinguishable from those of the clamped case 
with m = 1 (see figure |||), provided a < a c (u> = 1) ~ 0.138. For a > a c (uj = 1) we are back at 
m = 0, where no bifurcations were found to be allowed. 



3.3 The AT Instability 



The third and final type of transition to be analysed is the AT-line [18], where the relevant 
saddle-point of @ ceases to be replica-symmetric. Note that the fully recurrent limit u — » 1 of 
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a 



Figure 5: Location of the continuous replica symmetry breaking transition (AT line) in the 
(a, T) plane. For to = —1 the AT line collapses onto the line T = 0. 



the present model exhibits broken replica symmetry (RSB) near T = J2], whereas the u> — > — 1 
limit does not Q, which hints at the possible existence of an RSB transition at some critical 
value— 1 < to at < 1 f° r T = 0. In order to perform a bifurcation analyis a la [18], we go back to 
equation and work out the RSB saddle point equations: 



Qa/3 



Qa/3 



l. o27 o rV- ~2! 1. OT JDZ Z a ZR e^ JoZ - qZ 

—i j J x a XR\ > m..\ H — lapJo --, 



(29) 
(30) 
(31) 



with <x e {-1, l} n , z e W l , Dz = (2tt) 



-n/2 



e 2- 



J dz, and x Q = £ 7 [l-^Joq]^. Following [||] 



we now consider the so-called replicon fluctuations around the RS solution (|13|, 14 , 15|) : 

1 



Qa/3 -> b a /3 + q{t-5 a /3)+bQal3 



Qa/3 



-ia/3 [R5 a/3 +r(l-5 a j3)]+5q a j3 



(32) 



with the properties \5q a p\ <C 1, 5q a (3 = ^Q(3ai J2 a ^Qa/3 = 0, and 5q aa = 0. Working out the 



(coupled) variations in (^, 31 ) due to (|3^) , and requiring such variations to lead to an instability 
for n — > (a massless mode in first order perturbation theory), gives, after a modest amount of 
algabra, the condition 



1= [l-S-V fc cosh" 4 /? [MJ m+Jm)+V^]>£ 



(33) 
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The RS solution ( |l3| , p^JT5| ) is stable if the right-hand side of (33) is smaller than one. For Jq = 
(i.e. for the feed- forward model of || ||) clearly no RSB occurs. For Jq > (i.e. for uj > —1) 



we have to solve (^) numerically, in combination with the RS saddle-point equations (13.14,15]). 
For stationary states in large chains (i.e. (m,q,r,) = (m',q',r') and L — > oo), and pure states 
m ti = <^A> the solution thus obtained is shown in figure |B| for u G {—0.5, 0, 0.5, 1.0} (for oj = — 1 
the AT line collapses onto the line T = 0). Our first conclusion is that for T = the RSB 
transition occurs at uj = —1; any finite fraction of recurrence in the interactions apparently 
destabilises the replica symmetric solution. Secondly, although we cannot interpret the T > 
data in figure || directly in terms of the operation of the present model, the fact that, as we 
move away from the fully recurrent case uj = 1, the noise level T at which replica symmetry 
breaks decreases monotonically, suggests that we can be confident that for the present model, 
as for the fully recurrent case, replica symmetry breaking effects will be of a minor quantitative 
nature only. This, in fact, is borne out by the agreement observed between simulations and the 
RS solution. 



4 Discussion 

In this paper we have analysed a simple layered Ising spin neural network model, with a con- 
trollable competition between recurrent and feed-forward information processing. This model 
interpolates between the fully recurrent and symmetric attractor model of [|] and the strictly 
feed-forward model of [|], ||, §]. At zero noise level and in a stationary state, the model can 
be solved analytically near saturation, using replica theory (where we have restricted ourselves 
to the replica-symmetric ansatz), in spite of its interaction matrix being neither symmetric nor 
strictly feed-forward (which are the features on which analysis usually relies) . This property also 
turns it into a nice toy model for testing new tools for tackling the stationary states of neural 
network models without detailed balance. 

In two extreme limits, fully recurrent and fully feed-forward operation, respectively, the 
results of g g and {§ §, | are recovered correctly, as it should. In the intermediate regime, the 
built-in competition between recurrent operation (which is highly non-ergodic within individual 
layers) versus feed- forward operation (which is ergodic within layers), is reflected in a non-trivial 
way in various types of transitions. These describe saturation breakdown, simple ergodicity- 
breaking involving pure states, and the AT instability |l{J with respect to replic-symmetry- 
breaking. The largest storage capacity is found to be a c ~ 0.317, which is obtained for a specific 
balance of the two types of interaction. Replica symmetry turns out to break down as soon as 
one moves away from the strictly feed-forward limit, i.e. for any finite fraction of recurrence in 
the interactions. 

Our results, which are supported by numerical simulations, might also play a role in describ- 
ing the competition between recurrent and feed-forward operation in the peripherical regions of 
the brain (upon suitable quantitative adaptation of model details), where architectures similar 
to the one studied in this paper can be found. 
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